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Abstract: This research focuses on the results of the Flathead Watershed Delphi survey, 
a consensus-building methodology used to establish foundational knowledge, skills and dispositions 
for the Flathead Watershed Educators Guide, a place-based watershed curriculum for middle school 
grades based on the Flathead Watershed Sourcebook. Survey participants (n = 33) were chosen 
based on their expertise as educators, resource managers and scientists living and practicing in 
the Flathead Watershed in northwestern Montana, USA. Participants' responses were gathered 
through a three-round survey conducted by the Montana State University (MSU) research team 
using MSU's online course management system. Desire 2 Learn (D2L), an anonymous, asynchronous 
platform with distance accessibility. Round One responses gathered through the D2L discussion 
tool allowed participants to read responses and reply if desired. Round One discussion responses 
were reformatted into statements, which were then rated through two successive rounds using 
a 1-5 Likert scale. Of the initial 142 statements, 91 statements were retained in the final round. 
Final statements were cross-referenced with the Flathead Watershed Sourcebook to identify learning 
objectives for the Flathead Watershed Educators Guide. Final statements identified the knowledge, 
skills, and dispositions deemed most important for students in the Flathead Watershed to learn. 
Statements supported the need for place-based watershed education in fostering positive attitudes 
toward conservation and protection of the natural environment. 
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1. Introduction 

Delphi surveys have been used in a number of fields to determine, through consensus opinions of 
acknowledged experts, foundational practices within those fields. The Flathead Watershed Delphi 
survey determined, through consensus of knowledgeable community members, the knowledge, 
skills and dispositions considered most important for students to learn and know about the 
Flathead Watershed. 

To determine the curriculum foundations, conservation and education-minded community 
members living in the Flathead Watershed of northwestern Montana, USA, collaborated with education 
professionals in the Department of Education at Montana State University to design and implement the 
Flathead Delphi survey. The results show that survey responses in the foundational curriculum areas 
indicated strong consensus around place-based education principles valuing community awareness 
and service, ecosystem conservation and environmental stewardship. 

1.1. The Flathead Watershed 

A watershed is a complex system in which water withdrawals, land use, developments, 
and industries cumulatively affect all living organisms that rely on the river and watershed system. 
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These vital but conflicting ecosystem services result in tradeoffs between water quality, water quantity, 
ecosystem conservation and human benefits [1]. 

The Flathead Watershed has its headwaters in the pristine regions of Glacier National Park, 
the Great Bear Wilderness Area and the Bob Marshall Wilderness Area. The watershed of the Flathead 
River extends throughout a significant portion of northwestern Montana with one of its tributaries, 
the North Fork of the Flathead River, originating in the Canadian province of British Columbia. 
The North, Middle and South Forks of the Flathead River join with the Stillwater, Swan and Whitefish 
Rivers to feed Flathead Lake, at 191.5 sq. miles, one of the largest freshwater lakes in the Western United 
States. Flowing out of Flathead Lake, the lower Flathead River joins the Clark Fork River, after which 
its waters flow west into the Columbia River and eventually into the Pacific Ocean [2]. The Flathead 
Watershed is one of the largest, most biologically intact ecosystems in North America with over 
400 terrestrial wildlife species, including 11 amphibians, 11 reptiles, 319 birds, and 71 mammals [2]. 

Population in the Flathead Watershed is approximately 125,000 people. The Flathead Reservation 
of the Confederated Salish and Kootenai Tribes (CSKT) consists of over 1.2 million acres and comprises 
the southwestern quarter of the watershed. There are approximately 7753 enrolled tribal members, 
with about 5000 members living on or near the reservation (http://tribalnations.mt.gov/cskt). 
Population studies in the Flathead Watershed show that the land and the people living on it are 
inextricably bound together through economics and recreation. Rich natural resources, along with 
the dramatic natural beauty of the Flathead Watershed, create strong draws for people to move to 
and stay in the Flathead area. The quality of the natural environment is highly valued throughout the 
watershed and is considered a chief asset [2], People choose to live in the Flathead Watershed for what 
the land offers and they remain in the Flathead Watershed for cultural traditions of family and tribe. 
Figure 1 shows the geography and scope of the Flathead Watershed and its relative location in the 
western United States. 



Figure 1. Geography and location of the Flathead Watershed in western North America. 
Flathead Watershed topography and spatial extent; its relative location in Montana and within the 
Columbia River watershed in western North America. 
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1.1.1. Place-Based Flathead Watershed Education 

Watersheds are a successful organizing principle for curriculum. Everyone on earth lives 
within a watershed; the quality of life is greatly affected by the condition of the local watershed; 
and watersheds can serve as an instructional focus for active learning in science, mathematics, 
social studies, environmental education, and other subject areas [3]. 

Place-based education taught through the lens of a watershed recognizes that students are part of 
and can learn about their place within the natural and cultural boundaries of a local watershed. 

In a defining quote about the meaning and value of 'place' from the book: Places: Linking Nature, 
Culture and Planning, Nelson and Lawrence write: 

The way people understand and value places strongly influences decisions they make about 
their conservation and development. Places are more than mere locations. They consist 
of a broad set of natural and human processes and features, whose interactions and 
characteristics change through time. Bedrock, weather and climate, plants, animals, soils, 
human land uses, technology, institutions and social learning, or culture, vary among and 
define places as we know them [4] (p. 1). 

The attributes of the Flathead Watershed draw people who choose to live, work and recreate in 
and near its native ecosystems. Flathead Watershed residents active in organizations dedicated to 
conservation and education in the Flathead Watershed work to protect the relatively pristine quality of 
their surroundings. Place-based curriculum about the Flathead Watershed constructed with input from 
watershed community members creates a circle of connection to the watershed and to the students 
living there. 

1.1.2. Conceptual Framework for Place-Based Watershed Curriculum 

The theoretical framework of place-based education is exemplified in community-generated 
curriculum serving local students. The conceptual and epistemological framework that informs this 
research is the idea that place-based education develops closer regard and appreciation for the local 
environment and community. 

Placed-based education is the process of using the local community and environment as a 
starting point to teach concepts in language arts, mathematics, social studies, science and 
other subjects across the curriculum. This approach to education increases academic 
achievement, helps students develop stronger ties to their community, enhances students' 
appreciation for the natural world, and creates a heightened commitment to serving as 
active, contributing citizens [5] (p. 11). 

One of the basic premises of place-based education is that education should prepare people to live 
and work to sustain the cultural and ecological integrity of the places they inhabit [5]. In order to do 
that, Orr proposes that place-based education ensures that people become aware of ecological patterns, 
systems of causation, and the long-term effects of human actions on those patterns and systems [6]. 
Proponents of place-based education often envision a role for it in achieving local ecological and 
cultural sustainability [7]. 

The assumptions of place-based education find its roots in constructivist theory; the building 
of knowledge on known concepts and information. John Dewey, in his 'Pedagogic Creed' stated 
"all school life should grow out of home life... giving a background of past experience to the 
new ideas given in school" [8]. Constructivist educators contend that the background knowledge, 
previous experiences and fundamental worldview of students profoundly affect their interpretation of 
subject matter [9]. In support of that view, Gruenewald states: 

A multidisciplinary analysis of place reveals the many ways that places are profoundly 
pedagogical. That is, as centers of experience, places teach us about how the world works 
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and how our lives fit into the spaces we occupy. Further, places make us: as occupants of 
particular places with particular attributes, our identity and our possibilities are shaped [10] 

(p. 631). 

Gruenewald goes on to say that place-based education "aims to enlist teachers and students in 
the firsthand experience of local life and in the political process of understanding and shaping what 
happens there" [10] (p. 620). In an illustration of Gruenewald's understanding of the central role of 
place, Sobel writes of the Lubec High School in Lubec, Maine, where students in the school aquaculture 
study facility research the issues surrounding the historical fishing industry; actions that have led to 
revitalization of the threatened economic base of their town [5] (pp. 25-26). 

1.1.3. Community in Curriculum Development 

Smith discusses five thematic patterns in placed-based learning efforts: cultural studies, 
nature studies, real-world problem solving, internships and entrepreneurial opportunities, 
and induction into community process [11]. In his opinion, induction into the community process 
brings place-based education full circle in creating citizens who are educated within their community, 
by community members, and who then become acting community members themselves. Smith states 
that "The primary value of place-based education lies in the way that it serves to strengthen children's 
connections to others and to the regions in which they live" [11] (p. 594). Sobel concurs in his statement 
about community in his definition of place-based education: "Community vitality and environmental 
quality are improved through the active engagement of local citizens, community organizations, and 
environmental resources in the life of the school" [5] (p. 11). 

Place-based education has advantages over standardized curriculum through increased student 
engagement leading to greater student achievement [12,13]. A significant barrier to place-based 
education stems from standardized curriculum in school districts, measured through standardized 
testing imposed through federal law by the No Child Left Behind (NCLB) Act of 2001 [14]. 

According to Smith, "Because place-based education is by its nature specific to particular locales, 
generic curricular models are inappropriate in a place-based educational model" [11] (p. 587); an insight 
that points to the appropriateness of the local watershed concept to develop a place-based curriculum. 
This thought reinforces the validity of asking knowledgeable community members about what values 
they consider to be most important on which to build local watershed curriculum. 

1.2. Delphi Survey Research Method History and Use in Curriculum Design 

1.2.1. History and Nature of the Delphi Survey 

The Delphi survey is named for the Delphic Oracle who was active in the 5th century B.C. in 
the Greek city of Delphi. In Greece at that time, the Delphic Oracle served as an institutional process 
to help make policy decisions [15]. The oracle gathered information from a number of informants 
to deliver a truthful statement, compiled as a result of data from many sources [16]. Within the past 
century, the Rand Corporation first used the Delphi survey in the 1950s as a prediction method for the 
implications of the use of warfare technology on armament targets. Experts on the topic of nuclear 
weapons were consulted and their opinions pooled to formulate a prediction or consensus opinion [17]. 

Delphi surveys as census-building instruments can be characterized as "a method for structuring 
a group communication process so that the process is effective in allowing a group of individuals, 
as a whole, to deal with a complex problem" [18] (p. 3). Since Linstone and Turoff's study of the 
Delphi survey process, Delphi surveys have become widely used in more academic domains and for 
more purposes [19]. Delphi surveys, in seeking to uncover a reliable consensus, solicit the opinions of 
acknowledged experts in the field [20]. Knowledgeable participants can be experts in the field of study, 
as well as, being community members with personal stakes in the outcomes of the application of the 
survey results [21]. 
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A Delphi survey is an anonymous, iterative process, where initial responses are discussed and 
then ranked by individuals in the group as to their importance. This ranking process was initially 
described in the book. Group Techniques for Program Planning: A Guide to Nominal Group and 
Delphi Processes, Delbecq, Van de Ven and Gustafson [22]. Rated responses are winnowed down by 
group consensus determined by ranked scores to a core set of statements that participants feel are most 
valuable [18]. 

A Delphi survey is comprised of a minimum of two groups of individuals, the survey manager or 
managing team and the survey participants. A third interested party may include the entity for whom 
the data is being gathered [22]. The participants in the Delphi survey are chosen for their expertise 
in the area being studied or the complex problem being analyzed. Baker et al., in researching Delphi 
participants in nursing studies, discusses the definition of expert in terms of recognized knowledge, 
experience, and/or the ability to influence policy in the field [21]. An expert has been defined in 
various related ways, such as a group of informed individuals, specialists in their field, or someone 
who has knowledge about a specific subject [23]. In the matter of participants who are both willing to 
engage in the process of a Delphi survey and also considered experts in the subject area, Keeney et al. 
go on to say that: 

Simply because individuals have knowledge of a particular topic does not necessarily mean 
that they are experts. In fact, those who are willing to engage in discussion are more likely 
to be affected directly by the outcome of the process and are also more likely to become 
and stay involved in the Delphi. Hence, the commitment of participants is related to their 
interest and involvement with the question or issue being addressed [23] (p. 196). 

The same consideration is expressed by Moore, "A nuclear physicist is an appropriate expert if 
the Delphi concerns atomic energy and a resident of a neighborhood is an expert on what should be a 
community's goals" [24] (p. 51). 

Four key features define a survey as a 'Delphi': anonymity, iteration, controlled feedback, and the 
statistical aggregation of group response [19]. These features can offer specific advantages in a 
group decision-making process. Anonymity prevents domination by individuals, an advantage when 
participants may be hostile to each other, and it alleviates situations where personality styles would 
be distracting in a face-to-face setting [22]. Anonymity also contributes to the democratic nature of a 
Delphi [25]. Since a Delphi survey can be conducted online or at a distance, it is appropriate to use 
when personal contact is not possible due to distance, time and cost constraints. Online surveys can 
offer the advantage of asynchronous participation. 

The Delphi survey structure of iterative feedback, the repeated exposure of participants to 
statements generated by the group, develops insights that in the end are more than the sum of the 
parts [25]. Theoretically, the Delphi research process can be continually iterated until consensus 
is achieved. However, typically three iterations are adequate to collect sufficient data and reach 
consensus [26]. In a Delphi survey, feedback is controlled through the survey structure put in place by 
the survey manager. Clayton states that: 

The Delphi process aims to arrive at a level of consensus among the panel members. 
Consensus is assisted by the researcher providing feedback to each panel member of their 
previous rating together with a group measure of central tendency [27] (p. 382). 

Statistical aggregation of responses occurs through the analysis of ratings, often generated by 
Likert scale, to determine mean scores for items in the survey. 

The rigor of Delphi surveys, as determined by measures of reliability, validity, and trustworthiness, 
was researched by Hasson and Keeney in a study of nursing research Delphi surveys [28]. According to 
Hasson and Keeney, two characteristics of the Delphi survey create issues with establishing rigor. 
The first characteristic that challenges measures of rigor is the dual nature of classical Delphi 
surveys, depending on whether the survey is used for prediction or to explore and understand 
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situations. Hasson and Keeney express the duality in philosophy as "juxtaposition of a classical Delphi 
technique between positivist and naturalistic paradigms that raises a problem about which standards to 
adopt" [28] (p. 1696). The second characteristic of Delphi surveys leading to dilemmas around rigorous 
testing is the continual modifications of the method, with at least ten different recognized variants 
and applications of Delphi designs. While methods to establish rigor through quantitative means 
remains problematic, the qualitative measures of trustworthiness, obtained through the characteristics 
of the classical Delphi survey such as iterative rounds, the range and representativeness of participants, 
and detailed description of the collection and analysis process, are considered a more credible means 
of establishing rigor [28]. 

1.2.2. The Use of Delphi Surveys to Develop and Inform Curriculum 

Curriculum development results based on a Delphi survey depend on the participants chosen to 
respond, as well as, the initial prompts that begin the discussion. Depending on the goals of the survey, 
participants may be stakeholders in the survey outcomes. In curriculum development, stakeholders 
participating in a Delphi survey may be people invested in the community or profession who will 
be sharing responsibilities with future colleagues or community members trained in the proposed 
curriculum [29]. Delphi surveys have been found useful in curriculum development in business, 
nursing, medicine, agriculture, and technology along with environmental education. 

The Delphi survey has been used in curriculum development in several areas in environmental 
education. Project WET (Water Education for Teachers) is an example of curriculum development 
based on this approach. The necessary skills, knowledge and dispositions related to water and water 
resource education were determined by survey participants. The study sample of 268 included a 
range of people in natural resource management, management education, elementary, secondary and 
nonformal education, university researchers and natural resource agency managers representing every 
state in the United States [30]. Of the 5446 initial responses of the participants, 80 had a high level of 
agreement for the determining criterion; those responses became the framework for the Project WET 
curriculum [28]. Project WET is a widely respected curriculum used by school and natural resource 
educators in the United States and in many other countries (http://www.projectwet.org/). 

Archaeological Science for All (ASFA) developed a Delphi survey to establish a national 
framework that delineated the essential science concepts, skills, and dispositions that can be addressed 
in informal archaeology education activities [31]. The ASFA project consulted experts in the fields 
of informal science, archaeological science and archaeological education, choosing participants who 
were geographically situated across the United States. The ASFA Delphi was successful in reaching a 
consensus regarding archaeological content, skills and dispositions related to teaching science inquiry 
to under-represented populations. Results of the ASFA survey inform the development of a national 
plan to assist informal science educators throughout the nation [31]. 

2. Materials and Methods 

2.1. The Flathead Watershed Sourcebook and Educators Gidde Development Process 

Due to the changing use of land in the Flathead Watershed, and the very high value both long-time 
residents and new arrivals place on the natural characteristics of the environment, there is a strong 
concern in watershed communities that growth be balanced with environmental awareness and 
resource conservation. Concern for sustaining the unparalleled qualities of the Flathead Watershed in 
the face of increasing pressures for development led to undertaking the Flathead Watershed Delphi 
survey for the development of the Educators Guide. 

Developing awareness of the Flathead Watershed's unique natural qualities through education 
was the impetus for the creation of the Flathead Watershed Sourcebook and the proposed 
environmental education curriculum, the Flathead Watershed Educators Guide (Educators Guide) [2], 
The Sourcebook was researched and written by Fori S. Curtis, resident of the town of Whitefish in 
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the Upper Flathead Valley. Curtis was guided and advised by members of the organization known 
as the Flathead Community of Resource Educators (CORE). The CORE members who worked with 
Curtis are active in or employed locally with agencies and organizations such as the Forest Service, 
the Flathead Conservation District, the Flathead Lakers, the Whitefish Institute, Flathead Audubon 
Society, and the Flathead Basin Commission. Together with Curtis, the advisory group was known as 
the Steering Committee. 

With the completion of the Sourcebook, the Steering Committee planned the development of 
the Educators Guide as a set of curriculum material for middle school grades, with the Sourcebook 
being the source of content information. Following the advice of Dr. Michael Brody, Montana State 
University professor in science and environmental education, the Steering Committee chose the Delphi 
survey methodology to establish the foundation of the Educators Guide curriculum. The Steering 
Committee identified and contacted people who would be appropriate Delphi survey participants. 
Delphi survey participant selection was based on community members' expertise in areas pertaining 
to education and to the Flathead Watershed, such as local middle school teachers, natural resource 
scientists and managers and historians. Dr. Brody, Dr. Yates and Rose Vallor became the MSU research 
team for the Delphi survey and creation of the Educators Guide. 

2.2. Research Method and Design Appropriateness 

The Delphi survey method was chosen specifically for its consensus building capability. 
The Delphi survey research method was appropriate for this use because it created a circumstance in 
which knowledgeable stakeholders within the Flathead Watershed were able to provide input in an 
endeavor that may impact the education of their children about the watershed they live in and care 
about. As an anonymous and iterative consensus building tool, participants were able to learn about 
the views of other participants and respond with their own thoughts without pressure to conform. 

2.3. Validity and Reliability 

Descriptive validity is described and achieved by maintaining factual accuracy of the collected 
data [30]. Descriptive validity was ensured when researchers kept the original wording and intent 
of participants' responses to the initial Round One prompt in creating statements for Round Two of 
the survey. Table 1 is an example showing participants' responses from the knowledge section of the 
survey made into declarative, positive statements while preserving the intended concepts (participants' 
actual responses are italicized). 


Table 1. Example of creation of Round Two Statement from Round One responses. 


Round One response 

Round One response 

"I believe in the four C's- clear, connected, 
complex and cold. Those four things are 
essential for the bull trout, an essential animal 
of the Flathead Watershed", written by 
Participant 003. 

"Students seem to grasp the 4 C's quickly and are able to 
relate the concepts to the health of Big Creek. Students 
come back to the class room and create a presentation slide 
show illustrating the 4 C's and why they are essential 
components for the native bull trout", written by 

Participant 024. 

Combined statement for Round Two 

The four C's-clear, connected, complex and cold, are an essential concept for understanding of the Flathead 
Watershed, particularly for animals such as the bull trout, an essential animal of the Flathead Watershed, 
compiled by the MSU research team. 


Construct validity was established through internal measures, where it became evident that 
similar or identical themes were repeated throughout participants' comments. Participants then rated 
those same themes highly when evaluating them on Likert scales, resulting in those themes remaining 
in the final data collected in the Delphi survey. 
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As a further measure of validity, MSU researchers kept an audit trail of the research, starting with 
selection of participants [32], through initial prompts and responses and ending with the final rating 
and ranking of the Delphi survey statements. Information was captured electronically through the 
online course management system D2L and research team meeting notes were recorded and kept in 
hard drive storage. Conference call notes and email transcripts with Steering Committee members 
were also collected during the Delphi survey process. 

Reliability in this study is defined as the degree in which the research instrument consistently 
measures what it is designed to measure [33]. In Round Three of the Flathead Delphi survey, reliability 
was shown by the stability of scores achieved through asking participants to rerate the high ranking 
statements from Round Two for Round Three of the survey, creating a test-retest scenario for the high 
ranking statements in the survey. Table 2 shows the percentage of statements in each category that 
received a repeated measure of rating of important or very important (at or near 4.0) on the 1-5 Likert 
scale, with 5 being considered very important for students to learn. Table 2 also shows the inclusion 
of a fourth category of statements related to teaching that emerged from the analysis of participant 
responses from Round One. 

Table 2. Comparison of Number of Statements from Round Two to Round Three. 


Category of 
Response 

Number of Round 

2 Statements after 
Initial Comments 

Number of Statements 
Remaining after 

Final Ranking 

Cut-off Point in 
Likert Scale Ranking 
(1-5 Scale) 

% of Statements 
with 'Important' 
or above Rating 

Knowledge 

57 

36 

4.06 

65% 

Skills 

40 

19 

4.06 

48% 

Dispositions 

34 

27 

3.94 

79% 

Teaching 

12 

12 

3.94 

100% 


2.4. Participant Selection 

Enactment of a Delphi survey for the purposes of curriculum development required the 
participation of individuals with specialized knowledge and skills in areas for which information 
and opinions were collected. In the Flathead Watershed, the individuals identified were people who 
were knowledgeable about the Flathead Watershed in a wide range of fields such as natural science, 
economics, culture and history. Local educators were also considered important contributors to the 
process. Members of the Sourcebook Steering Committee and the CORE group formed the Curriculum 
Steering Committee (Steering Committee), who identified the individuals they felt would bring the 
needed experience to the survey. The Steering Committee decided on three categories of participants: 
educators, resource managers, and scientists. Members of the Steering Committee each contributed 
names of individuals that they knew from personal or professional contact within the watershed. 
The process is described here by one of the participants: 

All of us on the committee submitted suggestions, within the groups we 
identified... teachers, mgrs. and scientists... using our own knowledge of the community 
and asking people that participated in the original Sourcebook development. We wanted 
to get representation across the watershed [32], 

Decisions of who to invite to participate in the Delphi survey were made through consensus 
of the Steering Committee, who then issued invitations to the proposed participants. The initial 
number of respondents in the Delphi survey was 29 community members and 4 MSU researchers. 
Several members of the Steering Committee became survey participants. Participants in the survey 
came from a wide variety of backgrounds, but had common elements within their experiences. 
Those elements were engagement with the natural world, teaching experiences, typically in the 
sciences, a history of community service, and/or employment in natural resources infrastructures such 
as dams or fish hatcheries. Table 3 lists the affiliation and profession of the initial Delphi invitees. 
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Table 3. Profession and Affiliation of Initial Delphi Survey Invitees. 


Position 

Affiliation 

Educator 

Confederated Salish and Kootenai Tribes (CSKT) 

Educator 

Salish Kootenai College 

Educator 

Poison Middle School 

Educator 

Olney-Bissell School 

Educator 

Helena Flats School 

Educator 

Somers Middle School 

Educator 

Kalispell Middle School 

Educator 

Whitefish Middle School 

Educator 

West Valley School 

Educator—informal science 

Flathead National Forest 

Educator—informal science 

Flathead Lakers 

Educator—administrator 

Retired Teacher 

Resource Manager/Educator 

US Geological Survey/Flathead Lake Biological Station 

Resource Manager 

CSKT Wildlife Manager 

Resource Manager 

Kalispell Wastewater 

Resource Manager 

Rancher 

Resource Manager 

Creston National Fish Hatchery 

Resource Manager 

US Forest Service 

Resource Manager 

F H Stolze Land and Lumber 

Resource Manager 

Hungry Horse Dam 

Resource Manager 

Whitefish Lake Institute 

Resource Manager 

Flathead Conservation District 

Scientist / Educator 

Whitefish Lake Institute 

Scientist 

City of Kalispell 

Scientist 

Flathead National Forest 

Scientist 

Museum at Central School 

Scientist 

Flathead Lake Biological Station 

Scientist 

Flathead Valley Community College 

Scientist 

CSKT 

Investigator 

Montana State University 

Investigator 

Montana State University 

Investigator 

Montana State University 

Investigator 

Montana State University 


Final participation in Round One consisted of 25 people, down from the 33 initially invited. 
Final numbers of respondents in the first round in each category were nine educators, eight resource 
managers, and four scientists. In the subsequent rounds, five of the original participants either 
dropped out (2) or attempted to log on to the server carrying the survey and were not able to (3). 
Sixteen participants completed the third and final round. Table 4 shows the number of participants in 
each round of the Delphi Survey. 


Table 4. Number and Category of Participants in the three Delphi rounds. 


Professional 

Association 

Total Number 
Invited Initially 

Participants in 
Round One 

Participants in 
Round Two 

Participants in 
Round Three 

Flathead Watershed (FW) Community Members 

Educators 

12 

9 (75%) 

8 (66%) 

5 (42%) 

Resource Managers 

10 

8 (80%) 

9 (90%) 

7 (70%) 

Scientists 

7 

4 (57%) 

4 (57%) 

4 (57%) 

Total FW Participants 

29 

21 (72%) 

21 (72%) 

16 (55%) 

MSU Research Team 

4 

4 (100%) 

0 

0 
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2.5. Flathead Delphi Survey Process 

The three rounds of the Flathead Watershed Delphi survey were delivered through the online 
course management system known as Desire to Learn (D2L), now called BrightSpace. D2L was selected 
as the survey implementation tool due to its flexibility in supporting long distance, asynchronistic, 
and anonymous dialogue. Participants were assigned login ID numbers for D2L for the purpose of 
anonymity and confidentiality in the Delphi survey. 

The first round of the survey consisted of responses written by participants to the survey prompts 
and participant comments made to other participants' responses. In the following two rounds 
participants rated statements created from Round One responses on a 1 to 5 Likert scale based on their 
opinions of the importance of the statement. In the second round participants were able to comment 
about individual statements, but only the research team was able to view the comments. In the third 
round participants had only the option of rating the statements retained from the second round. 

2.5.1. Round One 

The first round asked that survey participants respond to three prompts, one for each category: 
knowledge, skills and dispositions. The prompts were: 

• What are the essential concepts/issues that students should learn related to the Flathead 
Watershed? (For example; water quality or fish species); 

• What skills should students develop that would help them learn about and live in the Flathead 
Watershed? (For example; reading maps or writing a letter to city council); 

• What attitudes and values should students develop regarding to the Flathead Watershed? 
(For example; appreciating indigenous perspectives or valuing diverse opinions about resource 
management). 

During Round One participants were able to read and respond to each other's comments. 
This allowed ideas to be expressed and developed through dialogue and discussion. Participants were 
able to write at length about their ideas pertaining to the prompts and to respond to posts made by 
other participants. No restrictions were given to the number or the length of replies or responses. 
See Table 5 for an example of Round One comments and responses. 

Table 5. Example of Participant Responses for Round One, Disposition Prompt. 


Title of Response: Appreciating Where They Live 

Response Created by Participant 021 on 28 February 2014 12:42 

Students should first be aware of the Watershed and the incredible gift they have been given in living within it. 
Many don't recognize that where they live is one of the most incredible places in the world. People migrate 
back to the Flathead more often than not after experiencing differences in other parts of the country or they 
experience a sense of awe when flying back into the valley. Getting them to appreciate where they live is 
paramount in order for them to buy into taking care of their environment. A big part of appreciating the 
Flathead Watershed will come through honing their observation skills. 

Response by Participant 001 2 March at 05:42 

I like the idea of explicitly creating learning opportunities that develop real awareness of this remarkable place. 
Sometimes when you see it and benefit from it every day, you take it for granted. Good point made. 
Response by Participant 025 3 March at 09:26 

I concur as well. We have an amazing place here. The watershed is as good as it gets ecologically in the 
Lower 48. So I always stress that first and foremost, before I get into the threats and concerns that we have 
about our aquatic ecosystem. 

Response by Participant 018 4 March at 06:29 

I agree. As a person who grew up here I can vouch for the fact that it's easy to not understand at a young age 
how fortunate we are to live in this place. When this is all you've ever known, you don't really have 
good perspective. 

Response by Participant 004 6 March at 18:56 

Appreciation is a hard thing to develop in a young person but it is worth trying. 
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During this initial round the research team was actively but anonymously participating to 
help encourage discussions. Researchers' responses typically were clarifying questions based on 
participants' contributions, or statements that explored or agreed with ideas that participants had 
already expressed. Round One was open to participants for 14 days, from 17 February to 2 March 2014. 
At the conclusion of Round One, participants' comments in the three categories, knowledge, skills and 
dispositions, were transformed into positive, declarative statements. Statements were constructed 
to preserve the original intent and wording whenever possible. Efforts were also made to eliminate 
redundancy. A total of 142 statements were formed from participant responses. During Round One 
comment analysis, the research team recognized the need to create a fourth category due to the number 
of statements related to teaching methods. Teaching was added as a fourth category in the survey for 
Rounds Two and Three. 

2.5.2. Round Two 

Round Two of the survey was designed to narrow the focus of topics and begin to come to 
a consensus. Using a quiz tool in D2L, participants were asked to rank the revised Round One 
statements using a 1-5 Likert scale; 5—very important, 4—important, 3—neutral, 2—unimportant, 
and 1—very unimportant. Participants were also encouraged to comment on and explain their 
answers. The research team did not participate in rounds two and three. Table 6 provides an example 
of comments from Round Two in the disposition category that a participant included with their rating 
of statement #19. 


Table 6. Example of a Round Two Statement with Comments. 


Statement #19. Indian Education for All is knowledge that students should have. 

Comment # 19: Native American values on water and its importance to life is powerful. 

Important to demonstrate how curriculum would meet Indian Education for All standards; 

There should be more resources/training available for teachers. I believe that some teachers are reluctant to 
teach Indian Ed. because they don't feel comfortable with the amount of knowledge they have on the subject. 

Students will connect better with the landscape if they understand that it has been important to people for 
thousands of years. The more they are taught that they are a part of the landscape just as Native Americans are 
and were, the more they (hopefully) will realize that this isn't a place for them to take for granted. Students 
need to understand that it is the SAME watershed. I think that somehow (perhaps because of elapsed time 
when Native Americans were the only ones here), students get the impression that Native Americans were 
part of a different world. 

I prefer having the Indian Ed for All mixed in with the rest, not taught separately. 


Round Two was open for participation for 14 days, from 17 April through 20 April 2014. 
During this round, participants were able to provide comments supporting their ratings. The values 
from the Likert scale ratings were not made visible to other participants. At the close of Round Two 
mean scores were calculated and the statements were ranked from most to least important. 

Rankings were examined for a natural break or gap in mean scores to determine which items 
would remain in the final round of the survey. Each category revealed natural breaks at different 
numeric values, with all of those values at or above the 3.94 range, indicating statements were 
considered important or very important. Participant comments were taken into consideration to 
determine if any of the lower ranking statements had strong appeal for participants. There were no 
comments that suggested reasons to keep lower ranking statements. After eliminating the lowest 
ranking statements, the remaining statements became the final Round Three materials. 

Supplementary Materials Table S7 in Supplementary Materials contains Round Two Delphi 
Statements in Ranked Order 
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2.5.3. Round Three 

Round Three participants responded to the remaining statements in the four categories by again 
rating the statements on a 1-5 Likert scale. In this final round participants were asked only to rate 
the statements, without commenting. The timeframe for Round Three was shortened from 14 days to 
5 days due to participant availability. At the conclusion of Round Three, mean scores were calculated 
from the Likert scale ratings and statements were placed in rank order. The results were again examined 
for natural breaks. The gap, or break, in mean scores appeared at approximately 4.0, slightly higher 
than the previous round. This also indicated that the participants had come to a consensus on what 
was deemed most important for the Educators Guide. 

The research team consulted with the Curriculum Steering Committee by teleconference to discuss 
the results, and it was decided by consensus to include a few selected statements from just below 
the cut-off point of 4.0, since those statements contained concepts such as education about Native 
Americans, and use of nature journaling, considered important to include in the Educators Guide and 
that were not represented in other statements. Due to the limited input from CSKT members, it was 
the opinion of the research team and the Steering Committee that education about Native American 
tribal knowledge and history be included in the final statements. The use of nature journaling as a 
teaching tool was considered by the educators on the team as an important pedagogical practice in a 
watershed curriculum. A total of 95 statements were retained after eliminations in Round Three. 

Supplementary Materials Table S8 in Supplementary Materials contains Round Three scores in 
ranked order with Means and Standard Deviations 

3. Results 

The Flathead Watershed Delphi survey sought to answer the following research questions: 

1. What is the essential knowledge, related to the Flathead Watershed, that students should learn? 

2. What are the essential skills, related to the Flathead Watershed, that students should be able to do? 

3. What are the essential dispositions related to the Flathead Watershed that students should have? 

4. What strategies could be used to guide Flathead Watershed curriculum? 

5. How do the Delphi survey results relate to the Flathead Watershed Sourcebook? 

Note: in the following paragraphs, statements taken directly from the Delphi survey are shown 
in italics. The statement-identifying letter and number are indicated in parentheses following the 
italicized statement. The complete list of Round Three statements in ranked order in each category can 
be accessed in Supplementary Materials Table S8. 

3.1. Research Question One 

What is the Essential Knowledge, Related to the Flathead Watershed, That Students Should Learn? 

This research question was addressed in the initial prompt from Round One of the Delphi survey 
in the category of knowledge: What are the essential concepts/issues that students should learn related 
to the Flathead Watershed? (for example; water quality or fish species). This prompt provided the 
greatest number of responses. Figure 2 shows the final knowledge statements in rank order with 
standard deviation (SD) bars indicating plus or minus one SD from the mean score. The shorter lengths 
of the bars on the highest-ranking statements indicate less SD from the mean and a higher degree of 
agreement of the importance of that statement. The increasing lengths of the bars indicate a wider 
range of opinion about the importance of that statement. In general, standard deviations are within one 
score range, indicating that participants were in fairly close agreement about the relative importance 
of each statement. 
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Figure 2. Knowledge Statement Means with +1 and —1 Standard Deviation. Number and order of the 

knowledge statements in the final round of the Delphi survey, ranked by mean with ±1 SD bars. 

Several knowledge statements stand out as having larger SDs than typical for their rank. The first 
statement is Kll: Students need to know the basic premise of a watershed , that it is where they hunt and fish 
and recreate, and work, and live, and eat and grozvfood and consume food, and that as a residt; they are also 
influencing the watershed with all their everyday actions, M = 4.50. This statement received two scores of 3, 
neutral on the Likert scale, 4 scores of 4, important on the scale, and 10 scores of 5, very important 
on the scale. The next statement with a larger SD is K4: Students should be aware that a major threat to 
the Flathead zvatershed is zvater quality degradation, predominantly nutrient and sediment additions caused 
by human activities. The Likert scores for this statement were as follows, 1—unimportant, 1—neutral, 
4—important and 10 very important. The third statement with a wide range of SD is K21: Students 
should be azvare of the storm water conveyance system and that storm drains flow directly into our water 
bodies without treatment, M = 4.19. Scores for this statement were 1—very unimportant, 1—neutral, 
7—important and 7—very important. K36 is a repeat of the statement K21. It received scores of 
1—very unimportant, 1—unimportant, 0—neutral, 9—important and 5—very important for its second 
appearance on the list, M = 4.00. Having similar scores each time this statement was rated indicates 
consistency in participants' decisions about the statements. 

In the knowledge category the top ranking statement, with a mean score of 4.69, was K2: Students 
should be knozvledgeable about the plants and animals of the zvatershed, including invasive species and their 
impact on the watershed. The next three statements have mean scores of 4.63. Those three statements 
are: Kl: Students shoidd be azvare zvhy the zvatershed is important to the species offish and zvildlife that utilize 
the watershed; K30: It is important for students to know how humans impact the zvatershed; K32: Students 
should understand the characteristics of a healthy zvatershed. The content of the top ranked statements 
indicated survey participants expressed the ideas that learning about animals, plants and watershed 
characteristics of the Flathead Watershed were important for students. Table S8 contains all the Round 
Three knowledge statements in ranked order. 

3.2. Research Question Tzvo 

What are the Essential Skills, Related to the Flathead Watershed, That Students Should be Able to Do? 

The Delphi survey prompt addressing essential skills was: What skills should students develop 
that would help them learn about and live in the Flathead Watershed? (For example; reading maps or 
writing a letter to city council). Figure 3 illustrates the ranking order for skill statements for Round 3 of 
the Delphi survey with error bars showing +1/ — 1 standard deviation. As noted with the knowledge 
statements, statements with the highest Likert scale ranking also show the smallest standard deviations, 
meaning survey participants were in closest agreement on the importance of those skills for inclusion 
in the Flathead Watershed Educators Guide curriculum. 
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Figure 3. Skill Statement Means with +1 and —1 Standard Deviation. Number and order of the final 
skill statements in the final round of the Delpi survey, ranked by mean with ±SD bars. 

All the statements in the skills category received a mean score rating of 3.94 or above with the 
exception of S5: "Nature journaling" or "field journaling" gives young students the tools they need to quietly 
observe what is going on around them. With a mean score of 3.69, field journaling ranked below 3.94, 
but in consultation with the Steering Committee, it was considered an important skill to keep and 
incorporate into the Educators Guide curriculum. 

3.3. Research Question Three 

What are the Essential Dispositions Related to the Flathead Watershed That Students Should Have? 

The Delphi survey prompt for the disposition statements was: What attitudes and values 
should students develop related to the Flathead Watershed? (For example; appreciating indigenous 
perspectives or valuing diverse opinions about resource management) In the category of dispositions 
the mean scores vary from M = 4.56 to M = 3.81, less than a full point difference from the highest 
ranked score to the lowest ranked statement. In Figure 4 the disposition statements are shown in 
rank order by mean score, with +1/ — 1 SD bars. The relatively flat mean score line and short range 
of distribution of SD bars in Figure 5 show consistent ratings for disposition statements in Round 3. 
SD bars become longer for most of the statements as the rank order of the statement becomes lower. 
Table S8 contains all the Round Three skill statements in ranked order. 



Figure 4. Disposition Statement Means with +1 and —1 Standard Deviation. Number and order of the 
disposition statements from the final Delphi survey round, ranked by means with ± 1 SD bars. 

An exception to that trend is Statement D3: It is important that students engage in life styles (choices 
and decisions) that protect and sustain a healthy Flathead Watershed. M = 4.06, that had the highest 
standard deviation, SD = 1.12. This statement was considered very unimportant by one Delphi survey 
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participant; the complete range of scores are: 1—very unimportant, 3—neutral, 5—important, and 
7—very important. 

The last three statements in the dispositions ranking had mean scores of 3.88, 3.88 and 3.81, 
respectively. These statements were included in the final statements in consultation with the Steering 
Committee because it was felt the ideas expressed in them were important relative to the Flathead 
Watershed. The statements are: D7: Salish, Kootenai and Pend d'Oreille history and cultural resources, 
such as films, books, have a tribal perspective and are important sources of information on science, history and 
contemporary information, including topics such as Indian Water Rights and native place names, which should 
be used in classrooms in the Flathead Watershed; D22: The major issues affecting zuatersheds today, such as 
water rights, toxic spills from rail or truck, toxic cleanup, industrial pollution, are important issues for students 
to learn about; and D18: It is valuable to know the Native American perspectives and to understand how the 
ecosystem has changed: such knowing could lead to a greater appreciation for the Flathead Watershed. The mean 
scores of the last three statements fall below the lower end of the standard deviation error bar of the 
first statement, indicating those statements have a lower rating of importance. Table S8 contains all the 
Round Three disposition statements in ranked order. 



Figure 5. Teaching Statement Means with +1 and —1 Standard Deviation. Number and order of the 
teaching statements from the final round of the Delphi survey, ranked by means with ±1 SD bars. 

3.4. Research Question Four 

What Strategies Could be Used to Guide Flathead Watershed Curriculum? 

The teaching statement was an emergent category created after Round One of the Delphi survey 
when it was determined that a number of comments contained specific suggestions on ways to 
teach knowledge, skills or dispositions in a watershed curriculum. Figure 5 shows the ranked 
teaching statements. 

The line for the ranked mean scores for the teaching statements trends slightly downward, 
while the SD bars do not show a strong trend to lengthening or shortening with descending rank. 
Several statements, such as Tl, T4, and T9, have slightly longer SD bars. Statement Tl: A great 
Socratic Seminar topic for students could be the statement: "Whatever happens to the water happens to the 
people", M = 3.94 received one score of 1—very unimportant, 5—neutral, 3—important, and 7—very 
important. Statement T4: Teaching about diversity can be accomplished through expression in subject matter 
teaching. An example is the multiple zvays water quality can be introduced: explaining ivhat scientists look for 
in their work, explaining the cultural importance of native fish and their waterways to indigenous people, or 
discussing various approaches to resource management of the same zvater body and zvhy each manager might 
choose this approach. M = 3.94, received scores of 2—very unimportant, 13—unimportant, 4—neutral, 
1—important, and 0—very important. Statement T9: Reading about non-fiction topics is a good zvay to learn 
about animals and the watershed and satisfy common core standards requirements for young students M = 3.69, 
received scores of 6—very unimportant, 8—unimportant, 5—neutral, 1—important, and 0—very 
important. Supplementary Materials Table S8 contains all the Round Three teaching statements in 
ranked order. 
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3.5. Research Question Five 

How Do the Delphi Survey Results Relate to the Flathead Watershed Sourcebook? 

The creation of the Educators Guide was the goal of the Steering Committee and the motivation 
for conducting the Flathead Delphi survey. To determine how the Delphi survey results related to the 
Sourcebook, the Sourcebook contents were analyzed and sorted by major content areas. An outline of 
the Sourcebook was developed with the Sourcebook contents delineated into chapters, sub-chapters, 
sections and topics in each sub-section. The final Delphi statements were then matched to the 
Sourcebook topics in each sub-section in a matrix format. The Sourcebook contents were continually 
checked throughout the process for accuracy in capturing Sourcebook topic content and to ensure that 
Delphi statement interpretations were suitably matched to relevant Sourcebook content. 

Twenty-three learning objectives were developed for the areas of greatest match between Delphi 
statements and Sourcebook content. The learning objectives sought to be inclusive of the maximum 
amount of both Sourcebook contents and Delphi statements. The Sourcebook Delphi Statement matrix 
was reviewed by the Steering Committee, who gave their approval of the scope and content of the 
learning objectives. The twenty-three learning objectives became the basis of the Flathead Watershed 
Educators Guide. 

4. Discussion 

The success of the Flathead Watershed Delphi survey in developing a foundation for the Educators 
Guide is demonstrated by the richness and depth of the statements that constitute the final round 
results. The initial responses to the Round One prompts were thoughtful, heartfelt opinions that 
indicated the value participants placed on stewardship of the environment and community in 
the Flathead Watershed. The Delphi survey methodology created the opportunity for community 
members to express their own concern for and commitment to the physical and cultural Flathead 
Watershed attributes. 

The effectiveness of the Delphi research methodology in surveying experts to determine 
a foundation for curriculum is corroborated by evidence from several studies on curriculum 
development from Delphi consensus statements [30,31,34,35]. Examples of Delphi studies come 
from Brody, who discusses developing a water education curriculum based on the expert opinion 
of educators and scientists from across the United States, and Miller, et al. whose study determined 
a training curriculum within a specified field of enterprise, based on the consensus of experts in 
the field [30,34]. Common features of the Delphi studies that contributed to their success were the 
identification of experts in the field, the advantages in conducting the survey online, and the consensus 
building aspects created from the initial open discussion and multiple ratings of survey statements. 

A unique and innovative element in the Flathead Watershed Delphi survey is the contributions 
of local community experts informing the Educators' Guide curriculum for the essential knowledge, 
skills and dispositions that the youth of their own community should acquire. Building a census of local 
experts' opinions about what they consider important for students to know about their communities 
and environment is an unexplored area in curriculum development. 

In the case of the Flathead Watershed Delphi survey, the experts, who are also community 
members, played a key role in the quality of the statements generated from their responses to the 
survey prompts and to each other. Local experts who are community members themselves have a 
stake in the stewardship of their environment and the successful continuation of their communities. 
In a sense, asking local experts' thoughts and opinions for curriculum for their youth is giving them 
the opportunity of passing on their knowledge and wisdom to educate and nurture the young people 
who will succeed them in management and conservation of the watershed. 

An outcome of organizing a Delphi survey among people with common concerns and interests 
was the creation of a social construct that may not have existed prior to the survey. In the case of 
the Flathead Watershed Delphi survey, participants became members of a local network sharing the 
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same concerns for the environment, of their watershed, and the education of the youth within the 
communities of the watershed. 

A further innovative aspect of the Flathead Watershed Delphi survey is using the concept of a local 
watershed as an organizing principle for the survey. While there are curricula focused on watersheds 
in general, and there are examples of curricula for specific watersheds in the United States such as the 
Colorado and Missouri available through Project WET, the Flathead Watershed Delphi survey's basis 
was the participation of local citizens responding from their knowledge of their own watershed. 

Although the survey included individuals from a wide range of backgrounds, it could have 
been enhanced by the participation of tribal members of the Confederated Salish and Kootenai Tribes 
(CSKT). Two tribal members responded to the initial prompts in Round One of the survey and their 
ideas and opinions were included in the declarative statements rated in the second and third rounds, 
though they did not participate in those rounds. The Salish and Kootenai tribal members comprise 
approximately 6% of the population of the Flathead Watershed and own nearly one quarter of the land 
within the watershed boundaries. The Flathead Watershed is part of the traditional native lands of the 
two tribes. The CSKT are actively involved in conservation efforts to preserve native species and to 
maintain their ways of living based on Flathead Watershed ecosystems. 

The contributions of more elementary and middle school teachers would have been welcomed as 
well. A total of 9 participating teachers drawn from the 68 schools within the watershed comprise a 
small representative sample. With the target audience of the Educators' Guide being middle grades, 
the additional input from teachers would have been valuable. 

While there is no further work being planned involving data from this survey at this point, 
any additions to the Flathead Watershed Educators Guide would likely benefit from further input 
from CSKT members and educators in the Flathead Watershed. 

There are several areas for further research related to the Flathead Watershed Delphi survey. 
One area to explore is further analysis of the data collected in the Flathead Watershed Delphi survey. 
A comparison of the responses and rating data for each sub-group of participants, educators, resource 
professionals and scientists, may provide additional insight. Trends in the responses for each sub-group 
could point to consistent behavior and choices for each group, though the small sample sizes of the 
subgroups limit the applicability of the analysis. Another area of potential future analysis of Flathead 
Watershed data would be to look at statements receiving higher rating in comparison to statements 
rated with lower scores. Patterns in comparisons may point to methodology issues or trends in 
participant choices. 

Two other potential research areas to consider are assessing the effectiveness of place-based 
watershed curriculum and examining the validity of the Delphi survey approach in gathering input 
from local communities to determine curriculum foundational concepts. One possibility for assessing 
curriculum effectiveness would be to pilot the Educators Guide in Flathead Watershed classrooms. 
Measures of effectiveness would need to be determined in such a pilot study. The validity of a Delphi 
survey as a viable option to determine a foundation for watershed place-based curriculums could be 
explored by conducting a community-based Delphi survey in a similar watershed and comparing 
results of the two surveys. 

5. Conclusions 

The Flathead Delphi survey to determine a basis for the Flathead Watershed Educators Guide 
was created to develop a consensus of opinions about the knowledge, skills and dispositions about 
the Flathead Watershed considered important by conservation-minded and environmentally-minded 
local resident watershed experts. The high ratings for the concepts expressed in the survey responses 
validated the hypothesis that there are foundational knowledge, skills, and dispositions that can be 
identified to form the basis for the Educators' Guide. 

This research study shows that surveying community members who are experts in aspects of their 
watershed results in a substantial pool of knowledge, skill and disposition statements, agreed upon by 
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consensus, to form a basis for a place-based watershed curriculum. In essence, the Flathead Delphi 
survey is a research tool to learn about and examine what motivates resident stakeholders to conserve 
and protect environmental attributes and causes them to expend time and energy on education through 
curriculum that enhances appreciation and protection of local watershed attributes. 

A local watershed-based curriculum embodies the concept of place-based education. 
A place-based curriculum based on the watershed concept, looked at in both biological and cultural 
terms, recognizes the interdependence of the people and the ecosystems within the physical parameters 
defined by the flow of water through the landscape. The goal of a place-based watershed curriculum 
is education to create awareness that leads to stewardship. 

The use of a Delphi survey for curriculum content development based on the attributes of a 
watershed is a new concept in education. Determining the values community members hold in 
relationship to the watershed they live in is an unexplored area of study in environmental education. 
Further research into the implications of the place-based education and community watershed concept 
is needed to understand and validate the approach. 

Supplementary Materials: The following are available online at www.mdpi.com/2227-7102/6/4/42/si. Table S7: 
Round Two Delphi Statements; Table S8: Round Three Delphi Statements. 
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